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Abstract 
In this technical report, implementation strategies from the first two years of Project ICEBERG 
exploration are described. Over this period of time, a new preschool assessment, curricular 
activity, and teacher training tool called the Learning Receptiveness Assessment-Greenhouse 
application, was iteratively developed, refined, and piloted to better support preschool children’s 
literacy, behavioral, and working memory processing skills and facilitate improved learning as 
they approach the transition to kindergarten and beyond. During the exploration phase of tool 
implementation, we gathered experts from the field of assessment and early learning to 
determine and evaluate areas of need and contextual fit such as the availability and allocation of 
tool-appropriate support resources, the readiness of the tool for use across diverse classroom 
contexts, and its fit and alignment with relevant state and community initiatives and support 
systems. We highlight associated lessons learned that can influence effective and sustained tool 
implementation, namely the importance of observing and studying variations in tool use in actual 
preschool classrooms over protracted periods of time and identifying gaps in stakeholder 


alignment, with the goal of helping researchers build on our experience in their own work. 
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Project ICEBERG Exploration: Using Implementation Science to Guide Preschool 
Reading Disabilities Prevention 

There is growing realization that to benefit from early learning investment, effective 
implementation strategies must be used in developing, piloting, and scaling up innovations. The 
development of innovations to support evidence-based approaches to teaching and learning in 
preschool is critical, but they must also be accepted, implemented as intended, and sustained 
over time (Kainz & Metz, 2016). To support such ends certain elements should be present across 
each stage of implementation (e.g., exploration, installation, initial implementation, and full 
implementation; Metz & Bartley, 2012): (a) expert teams to lead and monitor implementation, 
(b) data and feedback loops to support decisions for ongoing improvement, and (c) sustainable 
infrastructure to promote and grow capacity (Metz, Naoom, Halle, & Bartley, 2015). 

Implementation teams, composed of individuals with expertise in initiating and steering 
innovations (Metz & Bartley, 2012; Metz, Bartley, et al., 2015; Meyers, Durlak, & Wandersman, 
2012), (a) enact improvement and sustainability approaches to support core innovation 
components, (b) build capacity, (c) use data- and outcome-driven problem-solving, and (d) apply 
system change to support scaling-up (see Coffman, 2007). Data and feedback loops serve as a 
basis for engaging in structured ongoing evaluation, refinement, and improvement steps 
(Chinman, Imm, & Wandersman, 2004). Generally speaking, high quality and timely 
information is used to improve decision-making. Implementation infrastructure drives desired 
change (see, for example, Active Implementation Frameworks [AIF]; Fixsen, Blase, Naoom, & 
Wallace, 2009; Fixsen, Naoom, Blase, Friedman, & Wallace, 2005), building both general and 
innovation-specific capacity at the individual and organizational levels (Flaspohler, Duffy, 


Wandersman, Stillman, & Maras, 2008). For example, building general capacity through teacher 
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and leadership buy-in and a positive and facilitative culture and climate are integral for 
successful implementation because such approaches contribute to the people’s willingness to 
dedicate time, resources, and staff to implementing core components. Educator buy-in and 
positive school culture also increase people’s commitment to gaining parental support and 
collaborating across linked organizations. 

Researchers implementing innovations in educational settings can utilize principles 
underlying implementation science, namely the systematic use of research- and evidence-based 
practices to improve child learning, as a guiding framework. To develop tools for improving 
education outcomes is not enough; researchers must employ strategies to leverage expertise for 
ongoing improvement, use data to drive decision-making, and build human and organizational 
capacity necessary for sustaining and scaling up innovations in ways that support their specific 
project needs. Principles underlying the overarching elements ideally manifest across stages of 
implementation, with progress determined based on supporting empirical evidence. 

In general, the initial exploration stage focuses on “assessing the needs of the community, 
considering the possibilities for meeting those needs, judging the feasibility of different program 
models to meet the identified needs, and deciding on a plan of action and the resources needed to 
enact the plan” (Metz, Naoom et al., 2015, p. 11). By evaluating innovation need, its fit into 
existing systems, capacity for implementation, available resources, and readiness for scaling up 
across the field, researchers can adhere to principles underlying core elements while adapting 
them to fit project idiosyncrasies (Blase et al., 2013). Here again, it is not enough to simply 
document that an innovation or tool works—tesearchers must also ascertain how, where, and 
why the innovation is successful. Documenting the particular conditions under which an 


innovation is successful simultaneously sets the stage for implementation in similar contexts and 
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across diverse education settings. The purpose of this report is to document implementation 
strategies from the exploration stage of Project ICEBERG, a project funded to support data- 
based decision-making in preschool classrooms using a tablet-based assessment tool. 
Project ICEBERG 

The Office of Special Education Programs (OSEP) Stepping-Up Technology 
Implementation grant (CFDA 84.327S) funded Project ICEBERG (Intensifying Cognition, Early 
literacy and Behavior for Exceptional Reading Growth) to improve preschool data-based 
decision-making and prevent Reading Disabilities (RD) in 2015. Although research clearly 
supports the use of classroom-tools to measure developmentally important knowledge and skills 
in K-12 learning contexts, less research has focused on data-based decision-making (DBDM) 
processes—or the use of academic, behavioral and cognitive data to support instruction (see 
Tindal, 2013)'. Still less research has focused on how to successfully use DBDM in preschool, a 
critical transitional period in which children learn how to learn (Rimm-Kaufman, Pianta, & Cox, 
2000). Critically, effective preschool programs set the stage for future learning and can lead to 
long-term positive academic and behavioral outcomes in school through their effect on children’s 
kindergarten readiness (Claessens, Duncan, & Engel, 2009; Yoshikawa et al., 2013). 

However, a variety of unique challenges to successfully extending DBDM to preschool 
settings exist, including preschools’ decentralized nature and programmatic diversity (i.e., 
quality, type [e.g., structure; play-based/learning-based/mixed, home-based, community-based, 


religious institution-based, center-based]), a lack of consensus and availability of effective 


' Tindal (2013) synthesized over 30 years of classroom-based assessment and measurement 
research findings in K-12 learning contexts and outlines a program of future research designed to 
answer the need for systematic evidence-based trainings in data use and connected decision- 
making practices. 
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“curriculum” within the field, and loosely structured (or practically nonexistent) systems for 
professional development and training. Project ICEBERG sought to identify effective 
implementation strategies around a reliable and valid early screening tablet-based technology 
tool, called the Learning Receptiveness Assessment (LRA), which was designed to support 
preschool teacher DBDM and children’s “literacy receptiveness” for improving reading 
achievement outcomes in kindergarten and reducing the risk of persistent RD. 

When the project began, the LRA was a tablet-based screening tool that preschool 
classrooms could use to quickly identify which children were at greatest risk for exhibiting 
literacy, behavior, and cognitive processing (working memory) difficulties that would hamper 
learning if not addressed. Minimal training was required to administer the 15- to 20-minute 
tablet-based assessment—children were led through the pre-academic and working memory 
tasks via headphones, while teachers were guided to complete the brief behavior rating scale at 
their convenience through a simply-designed interface. Although the administration and 
functionality of the LRA operated as intended early on, more work was needed to develop 
supportive strategies and resources to enable teachers to engage in DBDM for taking sustainable 
actionable steps based on assessment findings, with minimal training and flexibility to “fit in” 
across diverse preschool settings. 

In the first two years of Project ICEBERG (2015-2017), the exploration stage of the 
project, researchers and partnering stakeholders engaged in project-adapted implementation 
strategies and activities. Over this period, our team of researchers, comprising expertise in 
assessment and DBDM, early childhood, technology, beginning literacy development, learning 
difficulties, and teacher professional development, adhered to principles underlying the key 


elements of implementation science, designing activities to help ensure successful LRA 
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implementation to support DBDM at the preschool classroom level. More specifically, through 
documentation of preschool educator and stakeholder perceptions and needs around early 
screening and associated support resources and observing the use of the tablet-based LRA tool in 
situ and over time, researchers strategized to create and customize project activities based on 
research-based implementation science. For example, based on information gathered, it became 
clear early on in the exploration stage that a stand-alone assessment (and reported scores) would 
be insufficient for facilitating effective DBDM among preschool teachers. 

Although teachers could administer the assessment with fidelity and obtain results, a gap 
existed in teachers’ ability to use the results meaningfully to effectively reduce children’s risk for 
RD in preschool classrooms. An expanded, scaffolded environment was needed to clearly link 
assessment-guided DBDM processes with preschool teachers’ “instructional” practices, 
including assessment score interpretation (e.g., Which children are at greatest risk for RD based 
on literacy, behavior, and cognitive processing difficulties?), determining what children need 
(e.g., How can children identified as at-risk be best supported so that weak areas are 
strengthened?), and taking effective actionable steps with reflection (e.g., How well are children 
benefitting from learning supports? What might need to be changed to better help them?). 
Substantial differences in the extent to which preschool teachers intentionally guide children’s 
learning emerged as an important barrier to DBDM: Without clear instructional anchors from 
which to monitor and evaluate children’s progress, one cannot adequately determine and act 
upon their learning needs. 

In light of existing diverse (and sometimes contentious) curriculum and philosophical 
approaches combined with limited professional development opportunities for sustainably 


implementing evidence-based practices (EBP), we explored and tested which EBP might be the 
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most feasible and useful (i.e., effective and acceptable to both teachers and children) to 
implement across the year. Thus, rather than offer teachers a “treasure box” of learning activities 
and interventions from which they would be required to select and implement, we organized and 
embedded the most viable approaches into a systematic set of curriculum activities that 
supported building knowledge and skillsets measured by the LRA as the means to provide basic 
instructional anchors. We intentionally designed learning activities to flexibly fit with diverse 
pre-kindergarten approaches and conditions, with the aim of providing a supplemental 
curriculum in which teachers could easily implement EBP without extensive training. This 
approach required that a careful balance between intervention intensity and practical feasibility 
be struck, based on a “real classroom” understanding of “what works”. Where possible, we 
repeated intervention strategies across months (e.g., within and across literacy and behavior 
activities) to support teachers’ “on the job” learning over time. For example, regular repetition of 
EBP teaching/learning strategies across different targeted letters of the alphabet, allowed 
teachers to become deeply familiar and facile with the logistics of activities implementation, 
while strengthening and applying DBDM skills needed to effectively enact literacy learning 
activities framed by children’s LRA and progress-monitoring results over time. 

Figure 1 displays the initial (top) and revised current (bottom) ICEBERG logic models. 
The models include teacher-driven actions (in gray boxes) and three intended outcomes (right of 
initial model) that have advanced Project ICEBERG work around identifying strategies for 
sustainable implementation of the LRA tool for DBDM. In particular, in the first year of the 
project we identified and incorporated four main development strategies to support preschool 
teacher DBDM processes when using the LRA: (1) identification of at-risk children in a clear 


“High Priority Needs” (HPN) LRA score report, (2) creation of whole-class learning activities, 
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with print-ready materials, comprised of high impact literacy and behavior self-regulation 
strategies embedded within systematic, easy-to-implement curriculum activities designed to 
support literacy receptiveness, (3) alignment of learning activities with opportunities for teacher 
monitoring and documentation of learning progress (in situ using a tablet-based Activity 
Checklist to facilitate ongoing DBDM practices) for children identified with HPN, and (4) 
development of teacher-friendly, proficiency-monitored training resources to educate providers 
about fundamental concepts and practices needed for high-quality effective tool use. 
The result was the development of the framing Greenhouse web-based application (LRA- 
GH) for use beginning in Year 2 of Project ICEBERG. While the intended outcomes of the 
project remained the same across Years 1-2 exploration (i.e., growing children’s literacy 
receptiveness through DBDM), we expanded the logic model in Figure 1 to incorporate the 
implementation strategies we adapted and employed. Based on teacher viewpoints and 
systematic observations of LRA-GH activities used in preschool classrooms (University of 
Oregon, 2015-2017a, b, c; University of Oregon, Fall, 2015-2017a; University of Oregon. 
Spring, 2015-2017a), teacher-driven actions (gray boxes) evolved in the following ways: 
e The LRA administration window widened from two seasonal time-points (winter- 
spring) to three (fall-winter-spring) over the preschool year just prior to kindergarten; 
e Rather than teachers determining early RD risk, the LRA system identifies children 
at-risk of RD (as High Priority Needs [HPN]) based on their LRA performance; and 
e Rather than selecting EBP strategies, preschool teachers implement whole-class 
intervention through EBP strategies organized into curriculum activities for 
monitoring the progress of children identified with HPN using an embedded Activity 


Checklist to reinforce ongoing evaluation of children’s learning support needs. 
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Essentially, in the revised current logic model (Figure 1, bottom) preschool teachers 
complete the online training with embedded proficiency monitoring. This online training is 
intended to help them understand the fundamentals of LRA-GH system and DBDM, including 
LRA administration, meaningful score interpretation and use, and associated literacy and 
behavior curriculum activities, and how system components work together to enhance literacy 
receptiveness (to reduce RD risk and improve kindergarten reading outcomes). Once trained, 
preschool teachers administer the LRA in the fall and begin implementing LRA-GH activities 
monthly with all children, while specifically monitoring and responsively supporting the learning 
needs of children identified with HPN. Using the Activities Checklist, teachers report on and 
evaluate the effectiveness of their implemented curriculum activities each day, adjusting their 
classroom practices accordingly. The LRA is administered at two additional seasonal time points 
(winter and spring). The process of implementing curriculum activities in the whole class setting, 
individualizing support for children with HPN, and evaluating the effectiveness of curriculum 
activities and classroom practices is ongoing over the course of six months. 

Summary and Report Context 

In this report we detail the implementation strategies and activities that we undertook 
during exploration and associate them with improvements and “lessons learned” to document our 
steps with the aim of helping other researchers build on our experience. After describing the 
preschool sites and classrooms participating in the first and second year of Project ICEBERG, 
we frame the technical report based on selected Implementation Science activities outlined in a 
planning tool developed by the Active Implementation Hub (AIH) of the State Implementation 
and Scaling-up of Evidence-based Practices Center (SISEP) and the National Implementation 


Research Network (NIRN) and adapted for the exploration stage of Project ICEBERG (Blase et 


Project ICEBERG Exploration 9 


al., 2013; see https://implementation.fpg.unc.edu/sites/implementation.fpg.unc.edu/files/NIRN- 


StagesOfImplementationAnalysisWhereAreWe.pdf). 


Intended Literacy Receptiveness Assessment (LRA) for Instruction 
Population System Components / Teacher-driven Actions 


Implement Greenhouse 
EBP curriculum 
activities to provide all 
children with solid 
literacy and behaviors 
for learning 

Participate 

in online 

training 
Provide individualized 
support and 
monitoring for children 
with HPN 


Administer LRA fall, winter, and spring; engage in ongoing activities learning progress 


Figure 1. Initial (upper) and current (lower) Project ICEBERG logic models, with changes in 
teacher-driven actions (gray boxes) based on lessons learned during LRA implementation in 
Years 1-2 exploration. HPN = High Priority Needs; EBP = Evidence Based Practices. 
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Project ICEBERG Preschool Sites 

In the first two years of Project ICEBERG, we worked with three preschool classrooms, 
housed in three distinct early childhood centers. All three served children in separate 
infant/toddler, preschool and pre-kindergarten classrooms and were located in western Oregon. 
In Year 1, we worked closely with one center (P1). In Year 2, we continued our work with the 
initial site and added two additional centers (P2 and P3). P3 discontinued participation early on 
in Year 2 due to staffing issues unrelated to the project. The two continuing centers (P1 and P2) 
were non-profit centers providing pre-kindergarten programming for children ages 4-5, with an 
average class size of 15 children (an approximate 1:8 teacher to child ratio). Participating 
classrooms each had a lead teacher and 1-2 support staff who were regularly present. 
Additionally, both centers used a strategic coupling of pre-academic- and play-based 
programming, and offered breakfast, lunch, and snacks during the day, included in the price of 
tuition. Both also used off-the-shelf marketed curricula and had experience using observational 
and site-developed assessments for identifying children’s developmental needs. 
Oregon Quality Rating and Improvement System: Preschool Ratings and Characteristics 

The two participating preschool sites in Years 1 and 2 were rated as 4-star programs by 
the Oregon Quality Rating and Improvement System (OR-QRIS), a state-based system designed 
to raise “the quality and consistency of child care and early learning program” (Western Oregon 
University, 2018), whereas P3, which discontinued their participation early in Year 2, was rated 
a 3-star QRIS program. The QRIS was originally developed in the late 1990s. Currently, all 50 
states and several U.S. territories have statewide or regional QRIS in various stages of 
implementation (i.e., fully implemented, implemented and under revision, or in a piloting phase). 


Originating in 2011, OR-QRIS expanded statewide in 2014 and evaluates early childhood 
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programs based on a set of 16 standards that resulted from a crosswalk between the Head Start 
Performance Standards, the National Association for the Education of Young Children 
(NAEYC) accreditation standards, the Oregon Early Learning and Kindergarten Guidelines, and 
the State of Oregon licensure requirements (National Center on Early Childhood Quality 
Assurance, 2018). Broadly speaking, the OR-QRIS, recently revised and now called QRIS- 


Spark, rates preschools based on: 


e Program administration and business e Health and safety, and 
practices, e Children’s learning and 
e Staff qualifications and training, development. 


e Family partnerships, 
Evaluation of child learning and development is, in part, based on the use of research-based 
curricula and valid and reliable early screening and assessment tools (Western Oregon 
University, 2018). Designed to provide a simple and clear process for evaluating Oregon early 
childhood programs, the QRIS-Spark system assigns participating sites a “star score” ranging 
from 2 (committed to improving quality) to 5 (excels in essential standards that support quality 
learning for all children). QRIS ratings are available free to the public through an online 
repository (see http://triwou.org/projects/qris). 

Four-star preschool programs exceed the Oregon QRIS-Spark standards. These sites were 
the target audience for LRA-GH implementation during the exploration stage because their 
practices most closely align with the DBDM goals of Project ICEBERG. Namely, four-star rated 
programs have many qualities important to implementing LRA-GH tablet-based assessment and 
curriculum activities designed to help identify and reduce the risk for RD. Unlike lower-rated 


two- and three-star rated programs, four-star programs: 
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e Conduct child developmental assessments at least twice during the year; 

e Use enhanced materials and curriculum that promote equity, diversity, and inclusion; 

e Use tools to individualize teaching strategies for each child; 

e Include daily whole group activities; 

e Have documented guidelines for adult-child interactions that support children’s 

learning, language, and concept development; 

e Provide direct instruction and support on children’s social skills; and 

e Require staff to complete regular self-assessments on their professional performance. 
Thus, because 4-star quality rated preschool programs consistently engage in early screening, 
demonstrate a willingness to assess, identify, and address individual child learning needs, 
including preparation for kindergarten entry, and are aware of the need for (and perhaps have 
greater opportunities to participate in) professional development—they possess qualities 
important to the core of the work in Project ICEBERG. Our use of the QRIS system ratings was 
intentional to control for particular contextual quality factors. We selected these sites as our 
target preschool population during the exploration stage so that we could more clearly identify 
necessary resources and strategies to support sustainable LRA-GH implementation in sites 
deemed likely to effectively engage in and adopt DBDM practices. 

In terms of screening and assessment, children enrolled in P1 and P2 are administered the 
Ages and Stages Questionnaire, 3" Edition (ASQ-3; Squires, Bricker & Potter, 2009), a widely 
used developmental and social-emotional screener co-completed by the parent(s) and the lead 
preschool teacher or administrator (after a period of classroom observation and familiarity). 
Results from the ASQ-3 help P1 and P2 staff identify and address possible developmental delays 


early on in the children’s preschooling. The lead teacher and staff in P1 and lead teacher and an 
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administrator in P2 also administered a formative assessment called the Teaching Strategies 
GOLD® (TS GOLD), a web-based assessment that evaluates children individually in 
developmentally appropriate facets of social/inter-personal, self-regulation, early literacy, early 
math, and fine motor skills (Herman & Tabors, 2010). While both sites had two official check- 
points at the beginning and end of the academic year, informal ongoing observations were also 
conducted across the school year to help initiate early intervention screening and supports from 
external community resources for children identified as being at-risk for developmental delays. 
TS GOLD data were collected using a tablet and computer in P1 and using a paper-pencil 
version in P2. Both sites tabulated results and shared them with parents at seasonal parent- 
teacher conferences, with full reports also provided to parents at the end of the school year. 
Participating Preschool Site 1 

Preschool site 1 (P1) was the first preschool to join Project ICEBERG beginning in fall 
2015. P1 is 501(c)(3) non-profit preschool and has been providing childcare care for over five 
decades. P1 provides part- and full-day child care and preschool programming to children aged 3 
months to five years, grouping children into “rooms” based on age and developmental stage 
determinations to meet child needs. LRA-GH implementation took place in the pre-kindergarten 
classroom with 16 children (12 full-time, and 4 part-time) aged 4-5 years over both years of 
exploration (2015-2017). The classroom was run by a female lead teacher with assistance from 
two support staff who remained in the classroom all day (except for scheduled breaks). The P1 
lead teacher had taught in a preschool classroom for 18 years at the start of the project. P1 took a 
“balanced approach” to child learning, incorporating both academic- and play-based activities. A 
typical day in P1 consisted of a combination of organized and free academic and play-related 


activities, group discussion circle, rest, and snacks and meals. Academic and play-based 
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activities centered around early literacy (e.g., alphabet/phonics), math (e.g., counting, numeracy, 
patterns), art and music, and sensory and tactile (e.g., clay and dough). Learning activities were 
typically child-driven, based on children’s needs and interests, rather than completely 
thematically-driven, and were drawn from Creative Curriculum”, advertised as a hands-on, 
project-based preschool curriculum. Behavioral support at P1 was provided based on 
collaboration between the lead teacher and support staff around strategies that combine 
individual child and classroom needs with the P1 vision and mission. Notably, neither computers 
nor tablets were a systematic part of P1’s instructional practices, although the lead teacher 
reported being adept with touch-screen technology and using such technology during some 
music activities (and to administer the TS GOLD). 
Participating Preschool Site 2 

Preschool site 2 (P2) joined Project ICEBERG in fall 2016, the second year of 
exploration. P2 is also is 501(c)(3) non-profit organization and has provided preschool services 
for over four decades. P2 provides part- and full-day preschool programming to children two to 
five years old, and recreational programming for elementary-age children from kindergarten 
through fifth grade before and after school, grouping children into “classrooms” based on age 
and developmental needs. Implementation of the LRA-GH took place in the pre-kindergarten 
classroom, with 14 children aged 4-5 years. The classroom was run by a female lead teacher and 
two assistants. Because the center is a converted house, the classroom served as both a main 
thoroughfare to outside play and the site’s kitchen, making the location of the pre-kindergarten 
class unique. The lead teacher at P2 had taught in a preschool classroom for 30 years. 

P2 used a “play-based approach” to child learning and actively discouraged the use of 


paper-based curriculum materials. A typical day in the P2 pre-kindergarten classroom started 
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with two mixed classrooms of younger and older children (aged 3-5) collectively grouped in a 
single room as they were dropped off at the center. Once moved to their preschool classroom, the 
children received a blend of mostly “free-choice” (child-directed) play, exploration, and 
learning-related social and academic activities based on the HighScope® curriculum, advertised 
as a hands-on, child-centered program for the development of cognitive, social, and physical 
skills. Learning activities took place in whole-class and small-groups and were organized around 
children’s interests that were often thematic and linked across weeks. Activities typically 
included a combination of storybook reading, gardening, art, and music and movement, with 
little use of “worksheet” activities. Playtime inside and outside was abundant and interspersed 
throughout the preschool day. Notably, at the beginning of their participation in Project 
ICEBERG technology, was not an integrated or routine part of P2’s assessment or curricular 
practices. Outside of taking and searching for pictures related to classroom activities or 
emergency use, smart phones and tablet devices were largely unused in the classroom prior to 
LRA-GH implementation. In a focus group, P2’s lead teacher emphasized her trepidation with 
computer- and tablet-based technology at the beginning of her Year 2 participation (University 
of Oregon, Fall 2015-2017a), and she completed the TS Gold using the paper version. 

Effective Implementation Activities During the Exploration Stage of Project ICEBERG 

As described earlier, the main focus of Project ICEBERG was to identify effective 

implementation strategies around the LRA early screening tablet-based technology tool. 
Although the overall desired outcome of Project ICEBERG remained the same across the two 
years of exploration, enhancing preschoolers’ literacy receptiveness to reduce the risk of RD and 
improve kindergarten reading outcomes (see Figure 1), the desired teacher-driven DBDM 


practices and the associated support and resources necessary to achieve that outcome expanded 
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based on important lessons we learned. Over the course of the exploration phase, results from 
on-site field observations (University of Oregon, 2015-2017b), beginning- and end-of-year focus 
groups (University of Oregon, Fall 2015-2017a; Spring 2015-2017a), beginning- and end-of-year 
teacher surveys (i.e., Implementation Needs and Implementation Assets and Challenges surveys, 
respectively; University of Oregon, Fall 2015-2017b; Spring 2015-2017b), and hour-long teacher 
interviews (University of Oregon, 2015-2017a) guided the development of new supports and the 
refinement of existing supports, and served as our multi-method sources for ongoing data and 
feedback loops. The initial LRA iteratively developed into the LRA-GH, an inclusive early 
screening and reporting, curriculum, and teacher training innovation designed to more 
comprehensively support preschool teachers’ DBDM in a cohesive manner with respect to their 
own classroom practices and targeted learning outcomes. 
Convening Experts to Drive Implementation 

Expert stakeholders in the fields of early childhood education, behavioral intervention, 
professional development, and implementation science were identified and recruited to help the 
research group drive development and sustainable implementation of the LRA (and subsequent 
LRA-GH system). The Advisory Board for Project ICEBERG included experts from academia 
and the local and regional early learning communities. We describe the involvement of key 
members below. An early childhood expert from the University of Oregon provided Project 
ICEBERG with ongoing project support and guidance, including tool development feedback to 
ensure that the LRA-GH components would be developmentally appropriate and adequately 
support preschool teacher implementation needs. A second Advisory Board member with broad 
academic and community-based expertise in early childhood intervention provided tool 


feedback, implementation guidance (locally and state-wide), and facilitated project activities. A 
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third Advisory Board member provided guidance related to implementation science practices 
(e.g., from NIRN and SISEP) for scaling up the tool. In addition, Project ICEBERG’s research 
group leader Dr. Leilani Saez served as a project liaison to the county’s Early Learning Alliance 
(ELA), a non-profit organization and regional Early Learning Hub of the broader Early Learning 
Division of the Oregon Department of Education (ODE) that builds collaboration between early 
childhood and K-12 educators, health care and social service providers, local businesses, and 
parents to improve outcomes for teachers and children. This connection with the hub and its 
members provided tool feedback, implementation guidance, and ongoing project support (e.g., 
site recruitment, large-scale survey dissemination). 
Analyzing Data to Determine Need and Prevalence of Need 

At the beginning of exploration, we conducted a statewide survey to learn more about the 
field’s readiness to implement a tablet-based screening tool to support and improve teacher 
DBDM in preschool settings. We sought professional opinions from Oregon early learning 
stakeholders to help us identify implementation strategies, including those related to obstacles to 
sustainable integration of the LRA tool in preschool contexts. The survey, entitled Innovation 
Needs and Solutions, consisted of a combination of constructed (open text box) and selected- 
response (multiple-choice) items and was administered online over a three-week window in 
October-November 2015, with no compensation provided for responding. A secure platform 
made data collection and analysis anonymous and efficient. In order to cast a “wide net” of 
possible stakeholder respondents, we used a non-probability sampling approach, inviting early 
learning professionals with whom the Advisory Board and the research group were familiar 
through direct email and encouraging them to invite their colleagues. In addition, we supplied a 


direct link to the survey through websites that the research group manages with significant 
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educator traffic (http://www.brtprojects.org and https://www.easycbm.com). In total, fifty-seven 
early learning professionals and stakeholders responded to the survey. 

While full survey results are published in a technical report (see Irvin, Pilger, Saez, & 
Alonzo, 2016), two key findings impacted LRA development and implementation during 
exploration and are important to note here. First, respondents almost uniformly agreed that early 
screening was critical to identifying and meeting the learning needs of struggling children. 
Second, although the importance of coupling early screening results with targeted/individualized 
instruction was clear, professionals also specified significant shortcomings in current preschool 
practice that resulted in inadequate support for children, including: 

e Gaps in screening and eligibility processes related to accurately and comprehensively 
identifying children at-risk of poor outcomes and/or learning disabilities; 

e Deficiencies in teacher assessment and instructional knowledge and skills—including a 
lack of resources to support professional improvement; 

e Inequitable access to high-quality preschool programming for at-risk children; and 

e Misalignment between preschool and kindergarten systems in terms of academic and 
behavioral expectations and goals. 

These findings are consistent with prior research, and they made clear the need to 
develop a tool that could easily identify children at-risk for learning difficulties, provide 
equitable teacher access to curriculum and monitoring strategies driven by EBP (i.e., without 
requiring extensive or specialized training), and that could help planfully bridge the 
developmentally critical transition from preschool to kindergarten with opportunities to learn and 
practice targeted skills (Reynolds, Magnuson, & Ou, 2010). In particular, the survey findings 


(and our later exploration work) highlighted a critical challenge: How to adequately address 
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identified literacy weaknesses and related classroom behavior and cognitive processing 
difficulties in preschool classroom settings that vary widely in their approach, motivation, and 
resources. Furthering this challenge is the relative newness of state-wide standard expectations 
for guiding high-quality practices, which are still in their infancy (e.g., the Oregon Early 
Learning Standards were released in 2016), and have yet to substantially improve wide 
disparities in practice and weak cohesion to kindergarten. 
Selecting Targeted Areas to Address Need 

Based on the obstacles to implementing a tablet-based early screening tool pinpointed by 
early learning professionals in the Jnnovation Needs and Solutions survey (Irvin et al., 2016), 
two main areas of need at the teacher and preschool levels were identified that, if addressed by 


the LRA-GH, would help preschool teachers effectively and sustainably engage in DBDM in 


their classrooms. These 


Teacher- and Preschool-Level Areas of Need and Associated 
vO dua anese or aeEd DBDM Development Strategies and Supports During Exploration 


. ‘ 1. Improving teachers’ earl fhe ELLEN ih 
are displayed in the Cn chan wee: ceed proficiency tool 
screening and instructional : : : 
knowiedse and stills HPN identification and 
table to the right, along reporting 
with the resultant 2. Supporting integration of a ER culum 
activities 
tablet-based assessment and ee ein check Natit 
supports noted earlier i i i 
uPP Coen ee embedded HPN child 
classroom practices eras 
(see Project ICEBERG pico eue 


section, above, pp. 6-7). Development strategies targeting these areas were driven by the key 
elements that should be present across each stage of implementation—leveraging expertise 
during implementation, ongoing data and feedback loops, and fostering sustainable 


infrastructure. 
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Identifying Practices and Interventions Matching Targeted Areas of Need 

For new innovations and intervention tools to be usable, sufficient detail must be clearly 
available to stakeholders and educators so that they are implemented as intended and sustainably 
within and across diverse contexts, and importantly, so that they can be further refined and 


improved upon over time as new knowledge is gained and additional capacity built during 


implementation through Principles, Values, Outcomes, and Essential Functional 
Features Guiding LRA-GH Implementation 


research-practitioner 
Principles | The LRA-GH provides (a) early screening 


partnerships (SISEP, assessment to instruction DBDM support, and (b) 
scaffolded resources for implementing EBP to 
2013). Thus, a key step reduce preschool children’s risk of RD. 


The LRA-GH system values the critical roles of 
the following in preschool classrooms: 


in Project ICEBERG 


e Assessment-guided DBDM practices 
Evidence-based classroom learning strategies 


around identifying the 


EBP and development Teacher “on-the-job” professional 
— r development 
wore WOME e Diverse forms of high-quality engagement 
the areas of need was e Teacher and child scaffolded support 
In this context, the LRA-GH values empirical 
detailing and research as a guide for evidence-based assessment 
and instructional practices. 
demonstrating the exact 
. Outcomes _ | Use of the LRA-GH will result in enhanced 
nature of the LRA (and preschool use of DBDM assessment, instruction, 
and evaluation practices to strengthen children’s 
resultant LRA-GH) tool pre-reading development and reduce RD risk. 
Essential e Low-cost, easy-to-implement with minimal 
for project partners and Functional need for extensive or specialized training 
Features Senet “43 
potential future partners “noire eee guiding 
instructional support and evaluation 
during Year 3 and from Flexible yet systematic, thematically organized 
approach to coexist with existing classroom 
the early childhood practices 


community during scale-up. The table to the right details the principles and values related to 
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teacher actions and intended teacher/child outcomes that define the LRA-GH and yield its 
essential features—details that framed development activities during exploration (SISEP, 2013). 

It is important to consider the information contained in the preceding tables relative to the 
LRA-GH logic models presented earlier in Figure 1, including the evolution to the revised 
current logic model during exploration. Together, information conveyed in the current logic 
model and two tables offer a cohesive picture of the LRA-GH system—the essential features, 
how they function, and the intended outcomes for preschool teachers and children. Additionally, 
it is important to couch these identified principles, values, and associated outcomes within the 
broader task of assessing contextual fit, or “the match between the strategies, procedures, or 
elements of an intervention and the values, needs, skills, and resources of those who implement 
and experience the intervention” (Horner, Blitz, & Ross, 2014, p. 3). The LRA-GH tool needed 
to be designed based on and comprised of research-driven and evidence-based assessment and 
instructional principles and supports that when implemented by preschool teachers could 
cohesively and flexibly integrate with their “everyday” routines and practices, while 
simultaneously and iteratively improving them. In other words, the match between the LRA-GH 
and preschool settings helps determine if the tool is effectively and sustainably implemented, 
overcoming contextual obstacles, and producing the desired teacher and child outcomes— 
improved DBDM practices in the preschool classroom, and enhanced literacy receptiveness with 
a reduction in the risk of RD and improved reading outcomes in kindergarten, respectively. 
Assessing Contextual Fit of the LRA-GH: Supports, Usability, Fit, and Capacity 

During the exploration stage, we adapted portions of The Hexagon Tool (see Blase, 
Kiser, & Van Dyke, 2013), a tool designed to guide implementation-informed assessment of 


innovations and intervention tools in education and other social sciences settings, to help 
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evaluate and improve relevant areas of contextual fit for the LRA (and ensuing LRA-GH) tool 
(Metz, Louison, Lanier & Looper, 2018). Subsequent to determining field needs and their 
prevalence regarding a technology-based screening tool through analysis of the Innovation Needs 
and Solutions survey of early learning professionals (1.e., the two main areas of need displayed in 
the left side of the table on p. 19), we devised and conducted development activities with P1 and 
P2 with the goal of evaluating and improving the supports, usability, fit and capacity of LRA- 
GH implementation (i.e., the resultant supports displayed along the right side of the table 
displayed on p. 19). The development activities associated with each of these four areas, in part, 
comprise contextual fit and are detailed in the following section. 

Supports. During exploration, it is crucial to understand the availability and allocation of 
existing (or the development of new) resources to allow for initial adoption and sustained 
implementation of an innovation or intervention tool in context (Blase, Kiser et al., 2013). Over 
the first two years of Project ICEBERG, resource-based development activities focused on 
facilitating the implementation of strategic evidence-based DBDM practices for making 
principled and meaningful use of LRA scores to reduce RD risk through technology and teaching 
supports. In Year 1, we focused on identifying for teachers “recommended” assessment 
administration models for sustainable practice (e.g., “pull-out” or “keep-in”, in which children 
are assessed outside or inside of their classroom, respectively), key classroom routines in which 
to anchor learning activities (and embed well-known EBP intervention strategies) to flexibly 
work across diverse sites, and crucial teacher supports for initiating DBDM practices within the 
preschool context. For example, weekly observations and interviews were conducted with P1’s 
lead teacher at the end of observed preschool days over winter and spring (University of Oregon 


2015-2017a, b). These observations and interviews focused on testing the feasibility, 
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acceptability, and effectiveness of implementing specific curriculum activities, by systematically 
examining curriculum activity features that worked best/worst, staff preparation time expended, 
the quality of teacher-child interactions, and the degree to which children demonstrated success 
in their emerging proficiency with targeted concepts and skills. 

We also queried the P1 staff about suggested changes, and additional resources needed. 
Through this focus on the team (and not just the lead teacher), we learned about the need to 
provide learning activity implementation supports that could be used by any adult, with or 
without training or familiarity with the LRA-GH tool. Classroom teachers may depart in the 
middle of an activity to go on break, leaving other adults (e.g., therapeutic specialists, parents, 
site directors, or teachers from outside the preschool classroom) to facilitate the activity. This 
need for flexible supports led to the development of Table Top resources that briefly provide 
activity facilitators with an activity’s purpose, directions, necessary materials, and suggested 
questions to facilitate child engagement and understanding. Findings from these observations and 
interviews were considered alongside P1’s responses to the end-of-year survey entitled 
Implementation Assets and Challenges (University of Oregon, Spring 2015-2017b), which 
helped the research team learn more about the utility and challenges surrounding the LRA after 
implementation in Year 1 and helped guide the development of support enhancements the 
following year. 

In Year 2 the LRA app dashboard was extended to include the Greenhouse application 
(i.e., the LRA-GH). Accessible through a user-friendly monthly calendar, the app enabled 
teachers to document planning for their “own days” to support cohesive and integrated monthly 
activity planning. Through the GH application dashboard, teachers could select any daily 


learning activity plan (composed of Circle, Story, Play, and two table activities), with nearly 
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identical design formats used for both literacy and behavior activities. Additionally, we launched 
the LRA-GH support website (https://lra-greenhouse.brtprojects.org/) to provide private access 
to the expanded LRA-GH app, printable curriculum activity and support materials, and teacher 
training modules from a secure location. Based on Years 1 and 2 beginning- and end-of-year 
focus groups and survey feedback (University of Oregon, Fall 2015-2017a, b; University of 
Oregon, Spring 2015-2017a, b), we also embedded online teacher supports, including curriculum 
activities, directly into the LRA app. From these data, we learned participating teachers preferred 
online supports over printed manuals and guides that would require them to reference physically- 
separated resources. In addition, we learned that in order for these online supports to be used, 
they had to be brief, easy to read, with a clear connection to classroom practices. 

Based on Year 2 feedback, we refined how support materials were organized (print- 
ready) on the website and added images of classroom activity set-up to help remind teachers 
about the unique features of each activity, to better support teachers’ learning about LRA-GH 
activities, and help facilitate how they coordinated their preparation. Toward the end of Year 2, 
the online teacher training tool was developed from earlier face-to-face training feedback and 
teacher coaching targets (i.e., areas in which teachers required follow-up support). We developed 
six modules to address teachers’ need for knowledge related to assessment (both administration 
and meaningful score use), strategic activities use, and learning receptiveness concepts (e.g., 
growing strengths in emergent literacy and behavior regulation, as well as supporting working 
memory processing for effective learning). Screenshots of LRA-GH features and images and 
videos of set-up and implementation in classrooms were key components of the training content, 
in response to teachers’ expressed desire for real-world visual supports for their learning 


(University of Oregon, 2015-2017a). Selected-response proficiency questions were embedded 
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throughout each module to document teachers’ developing knowledge and understanding, with 
feedback auto-generated based on correct/incorrect responses. 

In Year 2, we also worked closely with teachers to design the Activities Checklist, a 
support for documenting the ongoing learning progress of children with HPN. We shared drafts 
and discussed the design to ascertain its fit with preschool teachers’ needs and capacity to use it, 
and field-tested an early version in classrooms. We solicited teachers’ feedback about ideal data 
reporting for ongoing monitoring of children’s progress and DBDM processes such as 
individualizing activities and adapting their typical practices to meet the learning needs of 
children identified with HPN (University of Oregon, 2015-2017a; University of Oregon, Fall 
2015-2017a; University of Oregon, Spring 2015-2017b). Future development work will include 
the addition of another support, an implementation fidelity checklist, to help preschool program 
directors observe whether teachers are implementing learning activities as intended. 

Usability. The readiness of an innovation or intervention to be used and adapted across 
diverse contexts should be evaluated in exploration—including an examination of how feasible, 
engaging, and applicable essential features and associated support resources are for effective and 
sustained implementation (Blase, Kiser et al., 2013). Oftentimes, scarce financial and training 
resources are obstacles to adequately serving preschool children, in particular for identifying and 
meeting the learning needs of children with or at risk for disabilities (Barnett & Carolan, 2013), 
findings reinforced by Oregon early learning professionals completing the Innovation Needs and 
Solutions survey (Irvin et al., 2016). Thus, from the earliest stages of exploration, development 
and implementation focused on making the LRA-GH widely available at a low-cost, with limited 
demands on technology and professional development training to give the tool greater utility in 


preschool sites with limited and widely-varying financial and staff resources/support. For 
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example, wireless tablets were chosen as the mechanism to implement the LRA-GH due to their 
low cost (a high quality and reliable tablet can cost less than $200), portability, and ability to be 
handled and used in preschool classrooms easily. 

Additionally, aware that teachers have limited time, we designed online teacher training 
modules to be completed at teachers’ convenience. Each training module was designed to take 
30-45 minutes or less, with exit-and-return capabilities built in to address completion 
interruptions. When the minimum proficiency is obtained in all modules, teachers may print their 
completion certificate to earn Oregon early childhood training credits (an implementation need 
expressed by teachers and early childhood stakeholders; University of Oregon, 2015-2017a; 
University of Oregon, Spring 2015-2017b). While obviously a resource that supports initial 
adoption and sustained implementation of the LRA-GH, the teacher training developed in Year 2 
also serves to alleviate expensive and laborious in-person teacher training and modeling that was 
necessary during exploration, but would be cost and time inhibitive for the research group and 
preschools during scale-up of the tool. In addition, the user-friendly “on-demand” nature of the 
training allows teachers to revisit concepts and move at a pace that fits their learning needs. 

Based on teachers’ expressed desire to learn on a “need to know” basis, we developed 
three types of one-page “How To” support materials: Technology use (e.g., navigating the 
tablet), curriculum activities implementation (e.g., selecting quality environmental print), and 
assessment-related decision-making (e.g., interpreting scores for meaning). These guiding 
support resources helped to make the technology easier to operate, the preparation and enactment 
of curriculum activities manageable, and decision-making (both big and small) clearer. Teacher 
feedback indicated that they preferred this constrained approach because it enabled them to 


easily and repeatedly find the information when needed, and these resources were perceived as 
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more accessible (in terms of their interest in reading them) than if they had been combined in a 
single hard-copy or electronic manual (University of Oregon, 2015-2017a, b). 

Curriculum activities were also augmented to improve the utility of the LRA-GH tool by 
creating feasible, engaging, and learning-goal-targeted activities and supports. In Years | and 2, 
we engaged in improvement cycles to revise and refine learning activities based on weekly in- 
class observations and in-person teacher feedback gleaned from post-activity interviews (i.e., 
teacher approaches to and feedback about classroom implementation and child learning 
outcomes were documented). For example, based on observation and interview findings, we 
developed additional support resources (e.g., activity intentional phrasing, word lists, Table 
Toppers, and implementation guidance documents) to enable more effective use of our strategic 
learning activities given preschool teachers’ busy classroom environments. 

Based on high demands on teachers’ working memory that were repeatedly observed 
(e.g., forgetting what they had been doing or saying during activity implementation after an 
interruption), it became clear that we needed to develop supports for teachers to maintain the 
learning goal focus despite their near-constant mental multi-tasking. Suggested “intentional 
phrasings” for the activities, although initially viewed skeptically by teachers, were quickly 
embraced as a means to implement regularly-repeating activities as intended. Teachers reported 
this linguistic and memory support as an asset of the tool because it allowed them to quickly 
understand learning activity targets and more efficiently identify and address the needs of 
children needing extra support in “real time”. 

Previously described user-friendly Table Toppers also helped teaching assistants to 
implement activities as intended—allowing them to take greater ownership of children’s learning 


without training or direct supervision/guidance by the lead teacher. In Year 2, outside 
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interventionists were regularly observed using the Table Toppers to quickly adapt activity steps 
(e.g., modifying expectations for paper cutting), as needed, for children with disabilities, to 
enable their full inclusion. Combined, these developments were noted by teachers as useful tool 
features that made the LRA-GH easy to use (University of Oregon, 2015-2017a, b). 

In future development, we plan to create information technology (IT) support on the 
LRA-GH website to help teachers manage their practice and increase the tool’s usability, 
particularly for teachers who may be less comfortable with technology. As stated earlier, 
educators at P1 and P2 expressed varying degrees of experience and comfort with computer- and 
tablet-based technologies in their classrooms, including using such technologies to assess and 
deliver learning activities. Similar variation in experience and comfort was found by respondents 
in the Innovation Needs survey (Irvin et al., 2016). These responses mirrored barriers commonly 
cited to incorporating technology tools in K-12 classroom settings, including a hodgepodge of 
sometimes disconnected purposes underlying technology use, lack of support resources, negative 
or apathetic attitudes and beliefs, and limited knowledge and skills (for example, see Bebell, 
Russell, & O’ Dwyer, 2004; Hew & Brush, 2007). 

Encouragingly, despite varying levels of technology experience and fear, both P1 and P2 
effectively used the LRA-GH tool by the end of their project participation. It is important to note, 
however, that the efficiency with which the lead teachers in P1 and P2 fully implemented the 
tool was impacted by their differing degrees of initial comfort with technology. Their initial 
comfort with using a tablet also impacted the substance and amount of in-person teacher training 
and modeling we provided during exploration. 

Lessons learned during the exploration phase play an important role in the enhancements 


we have incorporated into the LRA-GH. For example, the availability of frequently asked 
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questions (FAQ) on the LRA-GH website, formulated using the feedback we gleaned from 
classroom observations, post-activity interviews, and beginning- and end-of-year focus groups 
and surveys, should help make the tool more usable for teachers who, like the lead teachers in P1 
and P2, have diverse backgrounds in terms of experience and comfort using technology to 
deliver classroom assessment and learning activities. Along these lines, IT support provided 
directly to preschool teachers by technical staff supporting the research group would be an 
important supplement to the FAQs. Such ongoing IT supports can offer explicit assistance for 
unique technical difficulties preschool educators have experienced when learning to use the 
LRA-GH tool. Participating teachers could submit questions and concerns to IT support staff 
through an online form or contact them directly via email. In short, FAQs would serve as 
prescient help for addressing difficulties already observed and those deemed more universal in 
nature, while “live” IT support would offer teachers assistance in resolving their specific 
concerns or problems—both making the LRA-GH more usable for diverse users. 

Fit. The fit of an intervention into and across diverse, and often hierarchical, contexts 
should also be strategically addressed in exploration—including its alignment with related 
initiatives, structures, and support systems (Blase, Kiser et al., 2013). A guiding theme of our 
work was that we wanted the LRA-GH to “fit” with existing practice, rather than replace it. 
Research-based innovations are not always sustainable in real-world practice, including those 
aimed at integrating technology into classroom practices (Hew & Brush, 2007). By respecting 
diverse preschool cultures, we sought to create conditions under which EBP would coexist with 
unique, high-quality approaches. This consideration was a critical factor as most preschools are 
businesses and tools should benefit, not undercut, the organization—a fundamental difference 


between preschools and K-12 public schooling. For example, preschools rely upon their 
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uniqueness to maintain a competitive edge within the market, and while the shared use of a 
standard curriculum or tool may help to ensure consistency in early childhood development, it 
should not replace what makes providers distinctive and useful for those they serve. Briefly 
mentioned earlier, one example of development work that bolstered the fit with our participating 
classrooms (P1 and P2) during exploration was the revised calendaring interface that blended 
and integrated LRA-GH activities with teachers’ everyday practices, including their “own days”, 
with printing capability available to help the lead teachers and their classroom assistants remain 
organized. This feature helped to make clear the supplemental design of LRA-GH learning 
activities and our aim to integrate them with preschool classrooms’ existing practices. 

Similarly, an overriding theme of our work was to make the LRA-GH friendly and 
accessible to diversely educated teachers—including those formally prepared through post- 
secondary coursework as well as those who have been “community educated” through 
experience in the field. As a result of different education and preparation backgrounds, preschool 
teachers may see their professional roles differently. Thus, we were particular about words we 
used to help aspects of the tool be widely understood and applicable. For example, after 
identifying five fundamental activities common in most high-quality preschool classrooms, we 
carefully renamed them within our tool to expand their use beyond particular site philosophies. 
Through preschool site and webpage visits we learned how preschool sites vary widely in how 
they reference these classroom routines (even when the activities are the same). For example, 
“circle” time for some preschool sites connotes a specific time for whole-group learning about 
weather, calendar, and songs, whereas in other sites, this “carpet” time is used as an opportunity 
to gather less formally for group conversation early in the day. By naming what we would 


consider a basic 15-minute common time for learning together in the morning “Explain” we 
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created a shared terminology for a particular activity that supersedes pre-conceived expectations 
about “circle” or “carpet” time. This naming convention provides teachers with the opportunity 
to decide how they want to best integrate “Explain” into their morning. In this way, we 
acknowledged and valued their unique identities while also encouraging them to become part of 
a broader community of practice through the use of the LRA-GH. 

More broadly, we have sought through our work on Project ICEBERG and the LRA-GH 
to align with state-level investment in equitably improving preschool to kindergarten outcomes. 
This investment includes the recent release of the Oregon Early Learning and Kindergarten 
Guidelines, the result of state-sponsored collaboration between a range of experts in higher 
education, early learning and K-12 practice, and community support agencies. These guidelines 
are designed to communicate well-defined expectations for what young children should know 
and be able to do as across development, including as they transition from preschool into 
kindergarten, in five interrelated domains: Approaches to Learning, Social-emotional 
Development, Language and Communication, Literacy, and Mathematics (ODE, 2017). 

Although currently a work in progress, we have preliminarily identified several areas of 
possible alignment between the LRA-GH and targeted learning outcomes in the Early Learning 
and Kindergarten Guidelines across the five development and learning domains (displayed in 
Figure 2). As providers work to align their practices with the Oregon Early Learning and 
Kindergarten Guidelines and the QRIS-Spark early childhood rating system, it is also important 
that new assessment, learning activity, and teacher training tools like the LRA-GH also align 
with this shared view for early childhood development. In doing do, such tools can be actively 
responsive to state-sponsored initiatives (like those in Oregon) that seek to formally align early 


learning and K-3 systems to improve reading and math achievement—including the development 
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of comprehensive assessment and data systems that link preschool with the early primary years, 
and early primary years with later public schooling (The White House Summit on Early 
Childhood Education, 2014). 

In future work, we plan to widen the net of possible preschool partners to explore the fit, 
utility and feasibility of the LRA-GH tool with other preschool settings. Future work, including 
in Year 3, will extend beyond QRIS-Spark 4-star rated centers (i.e., P1 and P2) to 4- and 5-star 
rated home providers, and include classroom settings with mixed ages (e.g., ages 3-5). Given the 
inherent range in programmatic diversity and the varying philosophical and curricular 
approaches of preschool sites in Oregon and beyond, examining the fit across such settings is 


critical to sustainable implementation and effective scaling up of the LRA-GH. 


LRA Greenhouse & Oregon's Early Learning Guidelines 
Coverage 


Mathematics 


Counting & Cardinality, 
Operations & Algebraic 
Thinking, Numbers in Base 10, 
Measurement & Data, Geometry 
& Spatial Sense 


AU Goals 1&2 
CS Goals 1, 2,83 
VGoals 1&2 


Figure 2. Areas of alignment between the LRA-GH and the Oregon Early Learning and 
Kindergarten Guidelines (ODE, 2017). 


Capacity. Studying the capacity to use an innovation or intervention as intended— 


including in ways that foster buy-in from providers, practitioners, and families to sustain, 
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improve, and broaden its use over time—is important during exploration (Blase, Kiser et al., 
2013). Alongside identifying 4-star QRIS-Spark centers as our target audience (1.e., intended 
users) during exploration, because of their screening and enhanced curriculum activities 
practices, in Year 2 we used pre-participation self-evaluations through a beginning-of-year focus 
group, Implementation Needs survey, and structured observations followed by one-hour 
individual interviews to evaluate sites’ readiness and capacity to implement the LRA-GH. An 
extension of these efforts under development is a formal Classroom Readiness Checklist that is 
comprised of essential elements (i.e., classroom arrangement and routines, teacher behaviors and 
instructional practices) that preschool classrooms should have in place before implementing the 
LRA-GH. The checklist will be included on the public facing page on the website and available 
to potential LRA-GH users to self-evaluate their readiness to implement the LRA-GH tool. 

In general, we have found that site buy-in largely depends on the center’s underlying 
philosophy and interest in intentionally supporting children’s transition to kindergarten in a 
developmentally appropriate way. Although financial and technological concerns played an 
initial role in centers’ buy-in and support for LRA-GH use, they mattered less than the degree to 
which curriculum learning activities were perceived as aligned with how much the center 
emphasized a play or pre-academic orientation. For example, centers focused more on academics 
wondered why activities didn’t teach all 26 letters of the alphabet, whereas mostly play-based 
centers wondered why teacher directed activities were necessary. In contrast, because of the 
time-saving design of the LRA and growing expectations for collecting and using assessment 
data to improve outcomes, the assessment component of the tool was fairly widely accepted by 
most centers we approached. Centers that described themselves as “balanced” in their approach 


were most receptive to trying and continuing with the LRA-GH at their site. 
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In addition, teacher buy-in involved facilitating an understanding among educators that 
they have the capacity and ability to prevent RD through their classroom practices. Based on our 
interview work, it became clear that preschool educators struggle to be seen by others as 
professionals (and not babysitters), and yet, most have fragmented (or no support for) systematic 
professional development to back them. To help address this need, we had a professional video 
made to explain the importance of the fundamental underpinnings of the LRA-GH system (i.e., 
pre-academic knowledge [literacy and early math], self-regulation behaviors, and thinking 
supports [working memory] and their association with learning difficulties in young children). 
This professional video serves as freely-available, public-facing information that is posted on the 
website (https://lra-greenhouse.brtprojects.org), and aims to improve buy-in and build capacity to 
implement the LRA-GH tool by helping to frame the critical professional role of pre- 
kindergarten teachers during children’s transition to kindergarten. The video demonstrates to 
preschool teachers how they can help to reduce RD risk by building children’s receptiveness for 
learning before they begin formal schooling. 

Important to future work regarding LRA-GH implementation capacity are inadequacies 
in Oregon’s early learning databases. Unlike the state’s K-12 databases, in which children are 
assigned a unique service set identifier (SSID), currently there are no unique identifiers for pre- 
kindergarten children. Thus, there is not yet a state-wide data system in place to facilitate the 
sharing of information from the LRA-GH to the kindergarten setting. The creation of a 
mechanism to link early learning and K-12 systems could substantially improve communication 
and collaboration between preschool and kindergarten educators. Such a coherent system could 
help identified child needs become more efficiently understood and met early in kindergarten. 


Through our work with teachers during exploration, we know that they intend to use LRA-GH 
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information despite this data system gap for parent conferences and to put individual score 
reports in child folders to be shared with kindergarten teachers. In anticipation of a time when 
the state might adopt unique child identifiers for preschool children, we have built placeholders 
for this information in the LRA-GH system, and are encouraged by teachers’ readiness to share 
obtained information with their children’s future teachers using their own means. 
Discussion and Future Work 

This project highlights several important lessons learned for implementing technology- 
based DBDM tools in diverse preschool settings. First, for research groups co-creating an 
innovation tool for use in preschool classroom settings, it is critical to spend significant time in 
actual classrooms to find a balance between the intensity of intervention and its feasibility for 
children and accessibility for teachers. In our case, effective use of the LRA-GH involved 
implementing EBP in order to make high-quality instructional decisions about assessment- 
identified literacy and behavior regulation weaknesses. Although preschool teachers already 
observe and support children’s learning needs in their regular practice, use of the LRA-GH 
entails using systematic and intentional EBP for addressing identified weaknesses, of which 
many preschool teachers may not know about. However, scaffolding of and adjustments to EBP 
must be “built in” to the tool for enabling their translation into sustainable practice without 
extensive professional development. These modifications are essential, however, because 
extensive professional development is largely inaccessible to diverse preschool settings. During 
Years 1-2 exploration, we spent approximately 200 hours in classrooms observing, interviewing, 
and working with the participating teachers to fine-tune LRA-GH implementation. We found 
that routinely spending time in the classrooms was necessary to comprehensively understand and 


“fit in” with classroom culture and practices both unique in a given setting and common between 
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classrooms. These observations also provided insight as a means to generalize needs and tool 
refinements to other settings. 

Regular time spent offers the framework for ongoing data collection and teacher feedback 
and allows room for iterative and ongoing tool refinement and improvement cycles. For 
example, initial drafts of children’s storybook making created challenging trade-offs between 
concerns about feasibility (e.g., cost and preparation time), literacy skill intensity (e.g., how to 
gradually increase demands yet create a meaningful “story”), and child responsiveness (e.g., 
interests and background knowledge, drawing comfort, and letter formation experience). The 
iterative revision of this activity, based on evidence-based and developmentally appropriate 
practices, was made necessary by observing children and teachers in their natural preschool 
classroom settings. Through trying out different versions of this activity over time, we were able 
to develop a learning activity that regularly engaged children because it was sufficiently hands- 
on, open-ended and child-directed, and required few materials while remaining aligned with the 
overall literacy learning targets. Observation and tool refinement in these instances allowed the 
LRA-GH to better fit with existing preschool culture and practices, while maintaining the 
integrity of the research-based EBP. 

Second, returning to the same site during the second year of exploration was important to 
observe and hone the LRA-GH tool and its implementation. Of course, every classroom year is 
different because the composition of learners changes. Documenting how these changes 
impacted tool implementation provided the research team with new considerations regarding 
acceptable use, adaptations needed to meet greater learner diversity, as well as changes in teacher 
implementation with more advanced knowledge and deeper understanding. For example, in Year 


1 at Pl none of the children were identified with a developmental disability. However, in Year 2 
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nearly 20% of the class had an identified disability in which children received early intervention 
support from outside community supports for cognitive, language, behavior, and/or motor 
delays. Compared to the previous year, this change in learner composition impacted how the 
teacher implemented learning activities unrelated to her greater skill because it introduced the 
presence of specialists in the classroom, required that appropriate adaptations be made “on the 
fly” for full inclusion, and changed the depth with which concepts were explored. Studying 
implementation over time at the same sites, in this example with the same lead classroom teacher 
(and mostly different children, including a greater number with identified disabilities), was 
critical for getting a “real” sense of if and how the tool is implemented as intended after the glow 
of initial participation has worn off. 

Last, there is an invisible challenge in aligning within and across preschool and 
kindergarten systems. There is systems-level incorporation of assessment in many high-quality 
preschools—trecall that QRIS-Spark requires 4- and 5-star rated programs to assess children’s 
developmental needs as part of their regular classroom practice. While “on paper” there appears 
continuity and movement by programs in the direction of administering research-based 
assessment tools, in reality, the actual use of assessment data to improve the quality of early 
childhood programming to strengthen children’s development is far messier. It appears there is a 
disconnect between the use and understanding of terminology, in which the actual meaning and 
importance of key teaching and learning concepts gets lost in the shuffle of maintaining quality 
ratings and actually delivering associated EBP. 

For example, the use of the term “curriculum” by preschool teachers varies widely to 
describe activities used in the classroom, from those intentionally organized with learning goals 


to those haphazardly pulled together from different sources with vague intentions. Similarly, 
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although tasked with “assessing” children’s development, very little attention has been paid to 
helping teachers understand how to systematically document and evaluate children’s needs and 
respond within the unique context of their classroom environment. That is, “high quality” early 
childhood centers are expected to engage in meaningful assessment of children’s development 
without supportive guidance for making judgments that are based on widely accepted and 
standard criteria (rather than a particular product’s conclusion by score calculation). In the 
absence of clear expectations for preschool development, the extent to which assessment can be 
meaningfully used to improve child outcomes is questionable and unfairly burdens teachers to 
“engage” in practices that they may not fully understand. Clearly, while QRIS-Spark ratings and 
the Oregon Early Learning and Kindergarten Guidelines encourage meaningful assessment and 
use of the data to improve teacher decision-making and child outcomes, high-quality and 
assessment-driven DBDM is not occurring despite documented widespread use of well-known 
and validated developmental assessments like the ASQ-3 and TS GOLD. The LRA-GH tool is 
specifically designed to help remedy this disjunction in systems alignment manifested as a 
disconnect in terminology use and the use of EBP. 

When using implementation science as a framework for scaling up the use of an 
innovation or intervention, researchers are forced to identify gaps in alignment between 
stakeholders—in the exploration phase of Project ICEBERG, this included state administrators, 
preschool directors, community support organizations, and preschool directors and teachers. A 
new tool, even one that is research-based, might not take hold in the context of broader state 


investment initiatives in which the routine use of EBP in classrooms is far from clear. 
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Conclusion 

The insights gained over the exploration stage of Project ICEBERG highlight the 
importance of having a strategic plan for having (and developing) the necessary supports to 
enable people with varying degrees of technology knowledge and comfort to have the confidence 
to adopt and effectively and sustainably implement new technology-based tools as intended. The 
lessons learned highlighted in this report emphasize a key guiding principle of our development 
work during exploration: The co-creation of an online assessment, curricular, and teacher 
training tool with early learning stakeholders and preschool partners that works cohesively 
within existing culture, systems, and classroom practices. Despite logistical challenges related to 
staffing changes and infrastructure issues (e.g., slow wireless connections at unexpected times), 
response to the LRA-GH has been overwhelmingly positive. 

We encourage researchers to engage in implementation science strategies as they begin 
the process of scaling up their innovation to more systematically identify needs to fulfill and 
barriers to address before moving forward, as well as field-recognized assets that may further 
validate the relevance and benefits of use. Although challenging to consider multiple and diverse 
stakeholders, varying contexts, and situational demands early in this process, it is through these 
multi-layered connections (from classroom to state administration) that tools can have their 


greatest, sustainable impact. 
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